2800 hi l1c backgrounds by subagonsouth · Pull Request #2937 · IMAP-Science-Operations-Center/imap_processing

subagonsouth · 2026-04-08T19:47:14Z

Summary

Implements background rate calculation for HI L1C PSET datasets. Adds _compute_background_counts() and updates pset_backgrounds() to compute background counts from direct events and convert them to isotropic background rates with uncertainties.

Key Changes

`imap_processing/hi/hi_l1c.py`

Added _compute_background_counts() function: Computes background counts by filtering direct events according to background configuration and binning into 3D array (epoch, calibration_prod, background_index)
Updated pset_backgrounds() function: Changed from accepting pre-computed background counts to computing them internally from L1B direct events, then averaging across spin bins to produce isotropic background rates
Updated generate_pset_dataset(): Loads background configuration CSV and passes exposure times to pset_backgrounds()

`imap_processing/tests/hi/test_hi_l1c.py`

Updated test_pset_counts() and test_pset_counts_empty_l1b(): Removed background_df parameter and background_counts assertions
Rewrote test_pset_backgrounds(): Updated to match new signature with L1B dataset and goodtimes, added required SPICE fixtures
Fixed test_generate_pset_dataset_uses_midpoint_time(): Updated mock to return exposure_times DataArray

Dependencies

Depends on #2935 (background utilities)

Related Issues

Closes: #2800

Copilot

Pull request overview

Implements IMAP-Hi L1C PSET background-rate computation by deriving background counts directly from L1B direct events, applying background configuration scaling/uncertainties, and wiring the new ancillary input through the CLI and tests.

Changes:

Added background configuration DataFrame accessor/utilities and an event-filtering iterator for background selections.
Updated HI L1C PSET generation to load background config and compute isotropic background rates/uncertainties from L1B DE + exposure times.
Updated CLI ancillary handling and adjusted/rewrote affected tests to match new function signatures.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
imap_processing/hi/hi_l1c.py	Loads background config and computes background counts/rates/uncertainties during PSET generation.
imap_processing/hi/utils.py	Adds `BackgroundConfig` accessor + shared filtering utilities and a background iterator.
imap_processing/cli.py	Updates HI L1C PSET processing to require both cal-prod and backgrounds ancillary inputs.
imap_processing/tests/hi/test_hi_l1c.py	Updates tests for new `hi_l1c`/`generate_pset_dataset`/`pset_backgrounds` signatures.
imap_processing/tests/hi/test_utils.py	Adds coverage for the new `background_config` DataFrame accessor.
imap_processing/tests/hi/conftest.py	Adds fixture path for the background configuration CSV.
imap_processing/tests/test_cli.py	Updates CLI test inputs to provide both required HI L1C ancillary files.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

imap_processing/hi/hi_l1c.py

imap_processing/tests/hi/conftest.py

Copilot · 2026-04-08T20:06:43Z

imap_processing/tests/hi/test_hi_l1c.py

+    # Call pset_backgrounds with the new signature
+    backgrounds_vars = hi_l1c.pset_backgrounds(
+        empty_pset.coords,
+        background_df,
+        hi_l1b_de_dataset,
+        hi_goodtimes_dataset,
+        exposure_times,
+    )
+
    assert "background_rates" in backgrounds_vars
-    np.testing.assert_array_equal(
-        backgrounds_vars["background_rates"].data,
-        np.zeros((n_epoch, n_energy, n_cal_prod, n_spin_bins)),
+    assert backgrounds_vars["background_rates"].data.shape == (
+        len(empty_pset.coords["epoch"]),
+        len(empty_pset.coords["esa_energy_step"]),
+        len(empty_pset.coords["calibration_prod"]),
+        len(empty_pset.coords["spin_angle_bin"]),
    )


test_pset_backgrounds now only asserts output variable presence and shapes. Since pset_backgrounds() was rewritten to compute counts, apply scaling factors, and propagate uncertainties, this test should include at least one numerical correctness/sanity assertion (e.g., rates constant across spin_angle_bin for isotropy, non-negative rates/uncertainties, and expected behavior when exposure is uniform/zero).

imap_processing/cli.py

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-08T21:22:34Z

imap_processing/hi/hi_l1c.py

+                len(bg_coords["calibration_prod"]),
+                len(bg_coords["background_index"]),
+            ),
+            dtype=np.int32,


_compute_background_counts currently allocates background_counts with dtype=np.int32. For long pointings / high event rates, background counts can plausibly exceed the 32-bit range and silently overflow. Prefer using an int64 accumulator (e.g., np.int64) for counts to make the rate/uncertainty computation safe.

Suggested change

dtype=np.int32,

dtype=np.int64,

imap_processing/hi/hi_l1c.py

imap_processing/tests/hi/test_hi_l1c.py

imap_processing/tests/hi/data/l1/imap_hi_90sensor-backgrounds_20240101_v001.csv

…und_counts

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-09T21:28:27Z

imap_processing/cli.py

+                # Create mapping from descriptor to path
+                anc_path_dict = {
+                    dep.descriptor.split("-", 1)[1]: dep.imap_file_paths[
+                        0
+                    ].construct_path()
+                    for dep in anc_dependencies
+                }
+
+                # Verify we have both required ancillary files
+                if (
+                    "cal-prod" not in anc_path_dict
+                    or "backgrounds" not in anc_path_dict
+                ):
                    raise ValueError(
-                        f"Expected exactly one ancillary dependency. Got {anc_paths}"
+                        f"Missing required ancillary files. Expected 'cal-prod' and "
+                        f"'backgrounds', got {list(anc_path_dict.keys())}"
                    )


anc_path_dict derives keys via dep.descriptor.split("-", 1)[1], which will misbehave or raise IndexError if the descriptor format differs (e.g., missing sensor prefix or additional hyphens). To avoid fragile parsing, prefer a safer normalization (e.g., strip leading "45sensor-"/"90sensor-" when present, otherwise keep descriptor as-is) or map using the file’s parsed name fields, and raise a clear ValueError when required descriptors can’t be determined.

imap_processing/hi/hi_l1c.py

lacoak21

Nice! This looks good.

subagonsouth requested a review from Copilot April 8, 2026 20:00

Copilot started reviewing on behalf of subagonsouth April 8, 2026 20:01 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

subagonsouth force-pushed the 2800-hi-l1c---backgrounds branch from e193b40 to 04f964f Compare April 8, 2026 21:16

subagonsouth requested a review from Copilot April 8, 2026 21:16

Copilot started reviewing on behalf of subagonsouth April 8, 2026 21:16 View session

Copilot AI reviewed Apr 8, 2026

View reviewed changes

subagonsouth self-assigned this Apr 9, 2026

subagonsouth added this to IMAP Apr 9, 2026

subagonsouth added 4 commits April 9, 2026 15:20

Add background calculation to hi l1c

4385d31

Add background ancillary dependency to Hi L1C in cli.py

20ff74f

Remove esa_energy_step and spin_angle_bin from calculation of backgro…

13fe97d

…und_counts

Copilot feedback changes

7b0dade

subagonsouth force-pushed the 2800-hi-l1c---backgrounds branch from 220dc38 to 7b0dade Compare April 9, 2026 21:20

subagonsouth requested review from Copilot, lacoak21 and sdhoyt April 9, 2026 21:22

Copilot started reviewing on behalf of subagonsouth April 9, 2026 21:23 View session

Copilot AI reviewed Apr 9, 2026

View reviewed changes

subagonsouth added 4 commits April 9, 2026 15:35

Add external data decorator to new test

a9d88e3

Remove external kernel requirement from test

381f1ca

Copilot feedback changes

afcc166

One more failing test due to external data fix

d9aaf26

lacoak21 approved these changes Apr 10, 2026

View reviewed changes

remove checks for mixed good/bad DEs b/c I see unexpected bad DEs in l1a

0f6582f

subagonsouth merged commit ea5ee40 into IMAP-Science-Operations-Center:dev Apr 10, 2026
14 checks passed

github-project-automation bot moved this to Done in IMAP Apr 10, 2026

subagonsouth deleted the 2800-hi-l1c---backgrounds branch April 10, 2026 20:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2800 hi l1c backgrounds#2937

2800 hi l1c backgrounds#2937
subagonsouth merged 9 commits intoIMAP-Science-Operations-Center:devfrom
subagonsouth:2800-hi-l1c---backgrounds

subagonsouth commented Apr 8, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 9, 2026

Uh oh!

Uh oh!

lacoak21 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

subagonsouth commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

imap_processing/hi/hi_l1c.py

imap_processing/tests/hi/test_hi_l1c.py

Dependencies

Related Issues

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lacoak21 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

subagonsouth commented Apr 8, 2026 •

edited

Loading

`imap_processing/hi/hi_l1c.py`

`imap_processing/tests/hi/test_hi_l1c.py`